Remote Sensing Image Super-resolution Research Articles

A spectral and spatial transformer for hyperspectral remote sensing image super-resolution

ABSTRACT Due to the generally low spatial resolution of hyperspectral images (HSIs), early multispectral images lacked corresponding panchromatic bands, and as a result, fusion methods could not be used to enhance resolution. Many researchers have proposed various image super-resolution methods to address these limitations. However, these methods still suffered from issues, such as inadequate feature representation, lack of spectral feature representation, and high computational cost and inefficiency. To address these challenges, a spectral and spatial transformer (SST) algorithm for hyperspectral remote sensing image super-resolution is introduced. This algorithm uses a spatial transformer structure to extract the spatial features between the image pixels and a spectral transformer structure to extract the spectral features within the image pixels. The integration of these two components is applied to HSI super-resolution. After comparative experiments with currently advanced methods on three publicly available hyperspectral datasets, the results consistently show that our algorithm has better performance in both spectral fidelity and spatial restoration. Furthermore, our proposed algorithm was applied to real-world super-resolution experiments in the region of China's Ruoergai National Park, and subsequently, pixel-based classification was conducted on the super-resolution images, the results indicate that our algorithm could also be applied to future remote sensing interpretation tasks.

Multiscale Residual Dense Network for the Super-Resolution of Remote Sensing Images

Multiscale Residual Dense Network for the Super-Resolution of Remote Sensing Images

TTST: A Top-k Token Selective Transformer for Remote Sensing Image Super-Resolution.

Transformer-based method has demonstrated promising performance in image super-resolution tasks, due to its long-range and global aggregation capability. However, the existing Transformer brings two critical challenges for applying it in large-area earth observation scenes: (1) redundant token representation due to most irrelevant tokens; (2) single-scale representation which ignores scale correlation modeling of similar ground observation targets. To this end, this paper proposes to adaptively eliminate the interference of irreverent tokens for a more compact self-attention calculation. Specifically, we devise a Residual Token Selective Group (RTSG) to grasp the most crucial token by dynamically selecting the top- k keys in terms of score ranking for each query. For better feature aggregation, a Multi-scale Feed-forward Layer (MFL) is developed to generate an enriched representation of multi-scale feature mixtures during feed-forward process. Moreover, we also proposed a Global Context Attention (GCA) to fully explore the most informative components, thus introducing more inductive bias to the RTSG for an accurate reconstruction. In particular, multiple cascaded RTSGs form our final Top- k Token Selective Transformer (TTST) to achieve progressive representation. Extensive experiments on simulated and real-world remote sensing datasets demonstrate our TTST could perform favorably against state-of-the-art CNN-based and Transformer-based methods, both qualitatively and quantitatively. In brief, TTST outperforms the state-of-the-art approach (HAT-L) in terms of PSNR by 0.14 dB on average, but only accounts for 47.26% and 46.97% of its computational cost and parameters. The code and pre-trained TTST will be available at https://github.com/XY-boy/TTST for validation.

Single Remote Sensing Image Super-Resolution via a Generative Adversarial Network With Stratified Dense Sampling and Chain Training

Single Remote Sensing Image Super-Resolution via a Generative Adversarial Network With Stratified Dense Sampling and Chain Training

Two-Stage Spatial-Frequency Joint Learning for Large-Factor Remote Sensing Image Super-Resolution

Two-Stage Spatial-Frequency Joint Learning for Large-Factor Remote Sensing Image Super-Resolution

MFFAGAN: Generative Adversarial Network With Multilevel Feature Fusion Attention Mechanism for Remote Sensing Image Super-Resolution

MFFAGAN: Generative Adversarial Network With Multilevel Feature Fusion Attention Mechanism for Remote Sensing Image Super-Resolution

Remote Sensing Image Super-Resolution via Multi-Scale Texture Transfer Network

As the degradation factors of remote sensing images become increasingly complex, it becomes challenging to infer the high-frequency details of remote sensing images compared to ordinary digital photographs. For super-resolution (SR) tasks, existing deep learning-based single remote sensing image SR methods tend to rely on texture information, leading to various limitations. To fill this gap, we propose a remote sensing image SR algorithm based on a multi-scale texture transfer network (MTTN). The proposed MTTN enhances the texture feature information of reconstructed images by adaptively transferring texture information according to the texture similarity of the reference image. The proposed method adopts a multi-scale texture-matching strategy, which promotes the transmission of multi-scale texture information of remote sensing images and obtains finer-texture information from more relevant semantic modules. Experimental results show that the proposed method outperforms state-of-the-art SR techniques on the Kaggle open-source remote sensing dataset from both quantitative and qualitative perspectives.

Remote sensing image super-resolution based on cross residual compensation mechanism and attention mechanism

Remote sensing image super-resolution based on cross residual compensation mechanism and attention mechanism

Landsat-8 to Sentinel-2 Satellite Imagery Super-Resolution-Based Multiscale Dilated Transformer Generative Adversarial Networks

Image super-resolution (SR) techniques can improve the spatial resolution of remote sensing images to provide more feature details and information, which is important for a wide range of remote sensing applications, including land use/cover classification (LUCC). Convolutional neural networks (CNNs) have achieved impressive results in the field of image SR, but the inherent localization of convolution limits the performance of CNN-based SR models. Therefore, we propose a new method, namely, the dilated Transformer generative adversarial network (DTGAN) for the SR of multispectral remote sensing images. DTGAN combines the local focus of CNNs with the global perspective of Transformers to better capture both local and global features in remote sensing images. We introduce dilated convolutions into the self-attention computation of Transformers to control the network’s focus on different scales of image features. This enhancement improves the network’s ability to reconstruct details at various scales in the images. SR imagery provides richer surface information and reduces ambiguity for the LUCC task, thereby enhancing the accuracy of LUCC. Our work comprises two main stages: remote sensing image SR and LUCC. In the SR stage, we conducted comprehensive experiments on Landsat-8 (L8) and Sentinel-2 (S2) remote sensing datasets. The results indicate that DTGAN generates super-resolution (SR) images with minimal computation. Additionally, it outperforms other methods in terms of the spectral angle mapper (SAM) and learned perceptual image patch similarity (LPIPS) metrics, as well as visual quality. In the LUCC stage, DTGAN was used to generate SR images of areas outside the training samples, and then the SR imagery was used in the LUCC task. The results indicated a significant improvement in the accuracy of LUCC based on SR imagery compared to low-resolution (LR) LUCC maps. Specifically, there were enhancements of 0.130 in precision, 0.178 in recall, and 0.157 in the F1-score.

MFPWTN: a multi-frequency parallel wavelet transform network for remote sensing image super-resolution

MFPWTN: a multi-frequency parallel wavelet transform network for remote sensing image super-resolution

ESTUGAN: Enhanced Swin Transformer with U-Net Discriminator for Remote Sensing Image Super-Resolution

Remote sensing image super-resolution (SR) is a practical research topic with broad applications. However, the mainstream algorithms for this task suffer from limitations. CNN-based algorithms face difficulties in modeling long-term dependencies, while generative adversarial networks (GANs) are prone to producing artifacts, making it difficult to reconstruct high-quality, detailed images. To address these challenges, we propose ESTUGAN for remote sensing image SR. On the one hand, ESTUGAN adopts the Swin Transformer as the network backbone and upgrades it to fully mobilize input information for global interaction, achieving impressive performance with fewer parameters. On the other hand, we employ a U-Net discriminator with the region-aware learning strategy for assisted supervision. The U-shaped design enables us to obtain structural information at each hierarchy and provides dense pixel-by-pixel feedback on the predicted images. Combined with the region-aware learning strategy, our U-Net discriminator can perform adversarial learning only for texture-rich regions, effectively suppressing artifacts. To achieve flexible supervision for the estimation, we employ the Best-buddy loss. And we also add the Back-projection loss as a constraint for the faithful reconstruction of the high-resolution image distribution. Extensive experiments demonstrate the superior perceptual quality and reliability of our proposed ESTUGAN in reconstructing remote sensing images.

Multi-scale feature fusion kernel estimation with masked interpolation loss for real-world remote sensing images super-resolution

ABSTRACT In recent years, the application of deep learning to remote sensing image super-resolution has achieved promising results. However, most deep learning-based methods are often trained on remote sensing datasets constructed by bicubic downsampling, and their recovery effects on real-world remote sensing images are often unsatisfactory. This is because the process of generating low-resolution (LR) images from high-resolution (HR) images to construct training data pairs (LR-HR) using simple bicubic downsampling cannot reflect the degradation process of real-world remote sensing dataset images. In this paper, we propose a multi-scale feature fusion kernel estimation with masked interpolation loss for real-world remote sensing images super-resolution (MFFILSR) to address this problem. MFFILSR is divided into two stages: degenerate kernel estimation and SR network training. In the first stage, we propose a multi-scale feature fusion kernel estimation network that can effectively fuse multi-scale information, making the estimated downsampling kernel closer to the degradation patterns of real-world remote sensing images. In the second stage, we introduce a masked interpolation loss during generator training, by masking the interpolation loss, the artefacts of generated images can be effectively reduced. Extensive experiments show that MFFILSR has satisfactory super-resolution reconstruction performance for real-world remote sensing images.

A lightweight distillation CNN-transformer architecture for remote sensing image super-resolution

ABSTRACT Remote sensing images exhibit rich texture features and strong autocorrelation. Although the super-resolution (SR) method of remote sensing images based on convolutional neural networks (CNN) can capture rich local information, the limited perceptual field prevents it from establishing long-distance dependence on global information, leading to the low accuracy of remote sensing image reconstruction. Furthermore, it is difficult for existing SR methods to be deployed in mobile devices due to their large network parameters and high computational demand. In this study, we propose a lightweight distillation CNN-Transformer SR architecture, named DCTA, for remote sensing SR, addressing the aforementioned issues. Specifically, the proposed DCTA first extracts the coarse features through the coarse feature extraction layer and then learns the deep features of remote sensing at different scales by fusing the feature distillation extraction module of CNN and Transformer. In addition, we introduce the feature fusion module at the end of the feature distillation extraction module to control the information propagation, aiming to select the informative components for better feature fusion. The extracted low-resolution (LR) feature maps are reorganized through the up-sampling module to obtain high-resolution (HR) feature maps with high accuracy to generate high-quality HR remote sensing images. The experiments comparing different methods demonstrate that the proposed approach performs well on multiple datasets, including NWPU-RESISC45, Draper, and UC Merced. This is achieved by balancing reconstruction performance and network complexity, resulting in both competitive subjective and objective results.

CFMB-T: A cross-frequency multi-branch transformer for low-quality infrared remote sensing image super-resolution

Aiming at the problems of low resolution and fuzzy details of infrared remote sensing images, paper proposed a multi branch super-resolution method based on wavelet decomposition and a dataset processing method based on real degradation process. To improve the super-resolution reconstruction effect, the modulation transfer function is used to add the degradation process of the real environment to the dataset at first, and based on this, CFMB-T is proposed. CFMB-T took SwinIR as the architecture, separated the frequency information of the infrared remote sensing image through wavelet decomposition, and introduced a degenerate prior to focus on the recovery of high-frequency information. Moreover, CFMB-T also introduced a recursive module to ensure the ability of feature extraction while reducing model parameters. The detailed experiments on degraded datasets show that CFMB-T achieves better reconstruction results than other super-resolution models. Compared with SwinIR, with limited data and training rounds, the PSNR and SSIM of CFMB-T are improved by 1.07% and 0.71%, respectively, over SwinIR on the 4x super-resolution task, and 3.28% and 0.09%, respectively, on the 2x super-resolution task. The results demonstrate its great potential for infrared remote sensing super-resolution reconstruction.

IESRGAN: Enhanced U-Net Structured Generative Adversarial Network for Remote Sensing Image Super-Resolution Reconstruction

With the continuous development of modern remote sensing satellite technology, high-resolution (HR) remote sensing image data have gradually become widely used. However, due to the vastness of areas that need to be monitored and the difficulty in obtaining HR images, most monitoring projects still rely on low-resolution (LR) data for the regions being monitored. The emergence of remote sensing image super-resolution (SR) reconstruction technology effectively compensates for the lack of original HR images. This paper proposes an Improved Enhanced Super-Resolution Generative Adversarial Network (IESRGAN) based on an enhanced U-Net structure for a 4× scale detail reconstruction of LR images using NaSC-TG2 remote sensing images. In this method, in-depth research has been performed and consequent improvements have been made to the generator and discriminator within the GAN network. Specifically, before introducing Residual-in-Residual Dense Blocks (RRDB), in the proposed method, input images are subjected to reflective padding to enhance edge information. Meanwhile, a U-Net structure is adopted for the discriminator, incorporating spectral normalization to focus on semantic and structural changes between real and fake images, thereby improving generated image quality and GAN performance. To evaluate the effectiveness and generalization ability of our proposed model, experiments were conducted on multiple real-world remote sensing image datasets. Experimental results demonstrate that IESRGAN exhibits strong generalization capabilities while delivering outstanding performance in terms of PSNR, SSIM, and LPIPS image evaluation metrics.

Hybrid-Scale Hierarchical Transformer for Remote Sensing Image Super-Resolution

Super-resolution (SR) technology plays a crucial role in improving the spatial resolution of remote sensing images so as to overcome the physical limitations of spaceborne imaging systems. Although deep convolutional neural networks have achieved promising results, most of them overlook the advantage of self-similarity information across different scales and high-dimensional features after the upsampling layers. To address the problem, we propose a hybrid-scale hierarchical transformer network (HSTNet) to achieve faithful remote sensing image SR. Specifically, we propose a hybrid-scale feature exploitation module to leverage the internal recursive information in single and cross scales within the images. To fully leverage the high-dimensional features and enhance discrimination, we designed a cross-scale enhancement transformer to capture long-range dependencies and efficiently calculate the relevance between high-dimension and low-dimension features. The proposed HSTNet achieves the best result in PSNR and SSIM with the UCMecred dataset and AID dataset. Comparative experiments demonstrate the effectiveness of the proposed methods and prove that the HSTNet outperforms the state-of-the-art competitors both in quantitative and qualitative evaluations.

Enhancing Remote Sensing Image Super-Resolution Guided by Bicubic-Downsampled Low-Resolution Image

Image super-resolution (SR) is a significant technique in image processing as it enhances the spatial resolution of images, enabling various downstream applications. Based on recent achievements in SR studies in computer vision, deep-learning-based SR methods have been widely investigated for remote sensing images. In this study, we proposed a two-stage approach called bicubic-downsampled low-resolution (LR) image-guided generative adversarial network (BLG-GAN) for remote sensing image super-resolution. The proposed BLG-GAN method divides the image super-resolution procedure into two stages: LR image transfer and super-resolution. In the LR image transfer stage, real-world LR images are restored to less blurry and noisy bicubic-like LR images using guidance from synthetic LR images obtained through bicubic downsampling. Subsequently, the generated bicubic-like LR images are used as inputs to the SR network, which learns the mapping between the bicubic-like LR image and the corresponding high-resolution (HR) image. By approaching the SR problem as finding optimal solutions for subproblems, the BLG-GAN achieves superior results compared to state-of-the-art models, even with a smaller overall capacity of the SR network. As the BLG-GAN utilizes a synthetic LR image as a bridge between real-world LR and HR images, the proposed method shows improved image quality compared to the SR models trained to learn the direct mapping from a real-world LR image to an HR image. Experimental results on HR satellite image datasets demonstrate the effectiveness of the proposed method in improving perceptual quality and preserving image fidelity.

Context-aware lightweight remote-sensing image super-resolution network.

In recent years, remote-sensing image super-resolution (RSISR) methods based on convolutional neural networks (CNNs) have achieved significant progress. However, the limited receptive field of the convolutional kernel in CNNs hinders the network's ability to effectively capture long-range features in images, thus limiting further improvements in model performance. Additionally, the deployment of existing RSISR models to terminal devices is challenging due to their high computational complexity and large number of parameters. To address these issues, we propose a Context-Aware Lightweight Super-Resolution Network (CALSRN) for remote-sensing images. The proposed network primarily consists of Context-Aware Transformer Blocks (CATBs), which incorporate a Local Context Extraction Branch (LCEB) and a Global Context Extraction Branch (GCEB) to explore both local and global image features. Furthermore, a Dynamic Weight Generation Branch (DWGB) is designed to generate aggregation weights for global and local features, enabling dynamic adjustment of the aggregation process. Specifically, the GCEB employs a Swin Transformer-based structure to obtain global information, while the LCEB utilizes a CNN-based cross-attention mechanism to extract local information. Ultimately, global and local features are aggregated using the weights acquired from the DWGB, capturing the global and local dependencies of the image and enhancing the quality of super-resolution reconstruction. The experimental results demonstrate that the proposed method is capable of reconstructing high-quality images with fewer parameters and less computational complexity compared with existing methods.

CGC-Net: A Context-Guided Constrained Network for Remote-Sensing Image Super Resolution

In remote-sensing image processing tasks, images with higher resolution always result in better performance on downstream tasks, such as scene classification and object segmentation. However, objects in remote-sensing images often have low resolution and complex textures due to the imaging environment. Therefore, effectively reconstructing high-resolution remote-sensing images remains challenging. To address this concern, we investigate embedding context information and object priors from remote-sensing images into current deep learning super-resolution models. Hence, this paper proposes a novel remote-sensing image super-resolution method called Context-Guided Constrained Network (CGC-Net). In CGC-Net, we first design a simple but effective method to generate inverse distance maps from the remote-sensing image segmentation maps as prior information. Combined with prior information, we propose a Global Context-Constrained Layer (GCCL) to extract high-quality features with global context constraints. Furthermore, we introduce a Guided Local Feature Enhancement Block (GLFE) to enhance the local texture context via a learnable guided filter. Additionally, we design a High-Frequency Consistency Loss (HFC Loss) to ensure gradient consistency between the reconstructed image (HR) and the original high-quality image (HQ). Unlike existing remote-sensing image super-resolution methods, the proposed CGC-Net achieves superior visual results and reports new state-of-the-art (SOTA) performance on three popular remote-sensing image datasets, demonstrating its effectiveness in remote-sensing image super-resolution (RSI-SR) tasks.

A contrastive learning-based iterative network for remote sensing image super-resolution

A contrastive learning-based iterative network for remote sensing image super-resolution